1. Field of the Invention
The present invention relates to a camera calibration method and apparatus for calculating a parameter representative of a characteristic of a camera, and more particularly to a camera calibration method and apparatus for calculating a parameter of a camera which is of a type picking up an image of a subject to output electronic image data.
More concretely, the invention relates to a camera calibration method and apparatus capable of accomplishing a stable estimation of a parameter with high accuracy on the basis of one picked-up image.
2. Description of the Related Art
With the recent progress of image processing technology, general-purpose computer systems, exhibiting high-level functions and great arithmetic processing ability, have come into widespread use, for example, among research organizations, enterprise offices and general homes. In addition, the computer application field has enlarged, and not only computer data but also other data including images and voices are translated into an electronic form computers can handle. For example, electronic image data captured through an image pickup means, such as a digital camera, and then read into a computer can diversely be processed through the use of computer resources for image combinations, image deformation and others.
Most of the existing cameras perform central projection through the use of a pinhole camera model. This central projection signifies that a color density of a point P on a surface of a three-dimensional object is put at the intersection between a straight line (also referred to as xe2x80x9cline of sightxe2x80x9d) connecting a projection center C with the point P on the object surface and a projection screen of the camera, and is for forming a projected image. In the case of the central projection, regardless of size identification of an object, as the object approach the projection center C of the camera, larger image projection takes place. On the other hand, as it recede from the projection center C, smaller image projection occurs.
Furthermore, it is obvious from the geometric optics that an image taken (photographed) from an oblique direction with respect to the front of a subject becomes a projection image obtained by the projection conversion of an image taken from a position just facing the front thereof. The fact that the projection image is obtainable by the projection conversion of the front image according to a projective transformation matrix H has been well known in the technical field of image processing. For example, if a front image is electronic image data captured through a digital camera, when the captured front image undergoes projection conversion through the use of a computer resource, it is possible to easily calculate a projection image, equivalent to when taken from an arbitrary direction (light of sight), at a relatively high speed. For example, xe2x80x9cUnderstanding of Imagexe2x80x9d 1990, written by Kenichi Kanaya and published by Morikita Shuppan, discloses that the original image is convertible into an image viewed at a different angle, through a projective transformation matrix.
The property on the geometric optics, related to the projective transformation, also applies to, for example, a method of measuring a distance from an object according to the xe2x80x9cstereo-methodxe2x80x9d. Here, the xe2x80x9cstereo-methodxe2x80x9d signifies a method of measuring the distances between points in a scene, that is, in a picked-up image, and the projection centers through the use of images taken from a plurality of station (view) points (projection centers) having predetermined positional relation to each other according to the so-called xe2x80x9ctriangulationxe2x80x9d principle.
In this specification, for convenience in description, the stereo-method will be conducted with two station points, that is, two cameras. One camera is used as a base camera, and is for picking up an image of a subject from a position right opposed to the front to output a base image. The other camera is a reference camera, and is for capturing an image of the subject from an oblique direction to issue a reference image. FIG. 10 illustratively shows the locations of a base camera and a reference camera with respect to a subject, and FIG. 11 illustratively shows a base image of a generally square pattern and a reference image thereof taken through the use of the base camera and the reference camera, respectively.
As FIG. 10 shows, a point P appears at the intersection nb between a straight line connecting a point P on a plane forming a subject with a projection center Cb of the base camera and a projection screen Sb of the base camera. The straight line for the connection between the point P and the projection center Cb of the base camera is a line of sight of the base camera. Additionally, a point P appears at the intersection nd between a straight line connecting the point P with a projection center Cd of the reference camera and a projection screen Sd of the reference camera. The straight line for the connection between the point P and the projection center Cd of the reference camera is a line of sight of the reference camera.
When undergoing a projective transformation, the line of sight of the base camera becomes the line of sight of the reference camera. The projective transformation is described with a projective transformation matrix H. The line of sight of the base camera is observed as a straight line on the projection screen of the reference camera, and this straight line is called xe2x80x9cepipolar linexe2x80x9d.
Furthermore, as FIG. 11 shows, a picked-up image taken by the base camera right facing the generally square pattern becomes square. An image taken by the reference camera viewing this pattern from an oblique direction, by contrast, appears as a trapezoidal form because of the reduction of a side at a longer distance from the station point. This depends upon the basic characteristic of the central projection that, regardless of size identification of an object, as the object approaches the projection center C of a camera, the effect is a projection of a larger image, and as it recedes from the projection center C, the effect is a projection of a smaller image.
As mentioned above, the picked-up image Id by the reference camera equals an image resulting from the projective transformation of a picked-up image Ib by the base camera. That is, the relationship between a point nb (xb, yb) in the picked-up image Ib by the base camera and the corresponding point nd (xd, Yd) in the picked-up image Id by the reference camera is given by the following equation, where H represents a 3xc3x973 projective transformation matrix.
[Equation 1]
nd=Hxc2x7nb
The projective transformation matrix H is a matrix tacitly containing internal parameters and external parameters of a camera and a plane equation, and has eight degrees-of-freedom because the degree of freedom stays in a scale factor. Incidentally, the xe2x80x9cUnderstanding of Imagexe2x80x9d 1990, written by Kenichi Kanaya (published by Morikita Shuppan), says that the corresponding points between a base image and a reference image is obtainable through the projective transformation.
The line of sight of the base camera appears as a straight line, called the xe2x80x9cepipolar linexe2x80x9d, on the projection screen Sd of the reference camera (refer to the above description and FIG. 10). The point P existing on the line of sight of the base camera appears on the same observation point nb in the projection screen Sb of the base camera, irrespective of the depth of the point P, that is, the distance thereof from the base camera. On the other hand, the observation point nd for the point P on the projection screen Sd of the reference camera appears on the. epipolar line in accordance with the distance between the base camera and the point P.
FIG. 12 is an illustration of a state of the observation point nd on the projection screen Sd of the reference camera. As illustrated in FIG. 12, as the position of the point P shifts from P1 through P2 to P3, the observation point in the reference image shifts from nd1 through nd2 to nd3. In other words, the position on the epipolar line corresponds to the depth of the point P.
When the observation point nd corresponding to the observation point nb of the base camera is searched on the epipolar line utilizing the above-mentioned property on the geometric optics, the identification of the distance of the point P is feasible. This is the basic principle of the xe2x80x9cstereo-methodxe2x80x9d.
However, the production of a perspective image based on a front image of a subject actually taken or the measurement of the distance to an object from a plurality of images, taken by a plurality of cameras, according to the stereo-method is on the assumption that the image pickup optical system of the camera has a property agreeing completely with the theory. Accordingly, there is a need to make a predetermined correction of an image taken actually. For example, in general, a lens of a camera has a distortion parameter(s), and an observation point appears at a position shifted from the theoretical point. Thus, not until a parameter peculiar to a camera is calculated to make a correction of image data according to the calculated parameter at the projective transformation that a precise projection image is obtainable from a front image, and that a precise depth measurement is feasible according to the stereo-method.
In addition to the lens distortion parameter, camera""s parameters involve internal parameters representative of a characteristic of a camera and external parameters indicative of a three-dimensional position of the camera. A method of calculating the camera""s parameters is generally called xe2x80x9ccamera Calibrationxe2x80x9d. Although many ways for the camera calibration have been proposed so far, nevertheless the settled way does not exist.
The most popular camera calibration method involves taking a calibration pattern comprising a plurality of reference points, whose positions in a three-dimensional space are known in advance, to calculate simultaneously all the camera""s parameters, such as internal parameters, external parameters and distortion parameters. This method is written, for example, in the Paper xe2x80x9cAn Efficient and Accurate Camera Calibration Technique for 3D Machine Visionxe2x80x9d (1986, IEEE) presented by Roger Y. Tsai. However, the use of this method requires the preparation of a calibration pattern in which reference points are shown accurately, and further requires a mechanism for positioning the reference points precisely.
In addition, another common camera calibration is a method of picking up an image of an object (for example, a bar or a cube) having a linear configuration. This method involves extracting points on an object from an image picked up to apply to a straight line by minimizing the error of the distance to a straight line comprising that point group, thereby calculating the degree of distortion. There is a problem which arises with this method, however, in that a possible error at the extraction of the points from the picked-up image affects the linear approximation and, ultimately, the calculation of the distortion. Accordingly, for obtaining the parameters stably and accurately, there is a need to take a plurality of straight line groups having a diversity of directionality, which causes the work to become complicated and the calculation amount to increase.
Still additionally, there is a method of conducting the camera calibration through the use of images obtained by photographing common scenes. For example, this method is written in the Paper xe2x80x9cAutomatic calibration and removal of distortion from scenes of structured environmentxe2x80x9d presented by Frederic Devernay or in the Paper xe2x80x9cLens Distortion Calibration Using Point Correspondencesxe2x80x9d presented by G. P. Stein. However, the calibration methods written in these documents require the extraction of linear components from a picked-up image or requires the determination of corresponding points to two or more picked-up images, so the present technical level does not reach stable estimation of parameters.
Moreover, since each of the above-mentioned methods relies on the extraction of local characteristic points for the parameter calculation, the extraction error occurs naturally. In other words, for example, there is a need to pick up a plurality of images for stabilizing the parameter estimation.
Accordingly, it is an object of the present invention to provide a high-performance camera calibration apparatus and method capable of achieving stable and high-accuracy parameter estimation for a camera of the type photographing a subject and outputting the corresponding electronic image data.
Another object of the invention is to provide a high-performance camera calibration method and apparatus capable of accomplishing stable and high-accuracy parameter estimation on the basis of one picked-up image.
For these purposes, in accordance with a first aspect of the invention, there is provided a camera calibration apparatus or method for calculating a parameter representative of a characteristic of a camera, comprising an image inputting means or step of inputting a picked-up image obtained by taking a photograph of a pattern with a predefined geometrical configuration through the use of a camera, an image holding means or step of holding a base image comprising the pattern with the predefined geometrical configuration, and a transformation parameter calculating means or step of calculating a transformation parameter on the basis of the association (correspondence) in pixel between the picked-up image and the base image.
The camera calibration apparatus or method according to the first aspect of the invention further comprises an image generating means or step of generating the base image comprising the pattern with the predefined geometrical configuration according to the definition. In this case, it is also appropriate that the image holding means or step holds the image generated by the image generating means or step.
Alternatively, the camera calibration apparatus or method according to the first aspect of the invention further comprises an image generating means or step of generating the base image comprising the pattern with the predefined geometrical configuration according to the definition and a light projecting means or step of projecting the generated base image on a generally plain plane. In this case, it is also appropriate that the image inputting means or step inputs a picked-up-image obtained by photographing the projected image, given by the light projecting means or step, through the use of a camera.
In addition, in the camera calibration apparatus or method according to the first aspect of the invention, it is also appropriate that the transformation parameter calculating means or step performs an image conversion of one of the picked-up image and the base image and further associates (maps) the converted image with the other.
Still additionally, it is also appropriate that the transformation parameter calculating means or step derives a projective transformation parameter, performs an image conversion of one of the picked-up image and the base image through the use of the derived projective transformation parameter and associates the converted image with the other to minimize a luminance (brightness) error between the corresponding pixels of both the images over the whole of the images.
Moreover, it is also appropriate that the transformation parameter calculating means or step derives a distortion parameter representative of a distortion factor of the picked-up image taking place at the image pickup by the camera and performs a projective transformation of the picked-up image from which distortion is removed through the use of the distortion parameter to associate the transformed image with the base image.
Still moreover, it is also appropriate that the transformation parameter calculating means or step corrects the luminance value of the base image in accordance with the luminance value of the picked-up image. In this case, a preferred correction is possible by extracting, from the picked-up image, an area in which the luminance values are approximately equal to each other and obtains the average value of the luminance values in the extracted area to replace the luminance value of the corresponding pixel in the base image with the average value.
Furthermore, a second aspect of the invention provides an image processing apparatus or method for processing a plurality of images picked up by a camera, with the apparatus or the method comprising image inputting means or step of inputting a picked-up image obtained by taking a photograph of a pattern with a predefined geometrical configuration through the use of a camera, image holding means or step of holding a base image comprising the pattern with the predefined geometrical configuration, transformation parameter calculating means or step of calculating a transformation parameter on the basis of the association (correspondence) in pixel between the picked-up image and the base image, and arithmetic means or step of obtaining the association in pixel among a plurality of picked-up images obtained by the camera through the use of the calculated transformation parameter.
The image processing apparatus or method according to the second aspect of the invention further comprises image generating means or step of generating the base image comprising the pattern with the predefined geometrical configuration according to the definition. In this case, it is also appropriate that the image holding means or step holds the image generated by the image generating means or step.
Alternatively, the image processing apparatus or method according to the second aspect of the invention further comprises image generating means or step of generating the base image comprising the pattern with the predefined geometrical configuration according to the definition and light projecting means or step for projecting the generated base image on a generally plain plane. In this case, it is also appropriate that the image inputting means or step inputs a picked-up image obtained by photographing the projected image, given by the light projecting means or step, through the use of a camera.
In addition, in the image processing apparatus or method according to the second aspect of the invention, it is also appropriate that the transformation parameter calculating means or step performs an image conversion of one of the picked-up image and the base image and further associates the converted image with the other.
Still additionally, it is also appropriate that the transformation parameter calculating means or step derives a projective transformation parameter, performs an image conversion of one of the picked-up image and the base image through the use of the derived projective transformation parameter and associates the converted image with the other to minimize a luminance (brightness) error between the corresponding pixels of both the images over the whole of the images.
Moreover, it is also appropriate that the arithmetic means or step performs, for the association between two picked-up images obtained by the camera, a coordinate transformation using a projective transformation for associating one of the two picked-up images with the base image and an inverse transformation of a projective transformation for associating the other picked-up image with the base image. In this case, the transformation parameter calculating means or step derives a distortion parameter representative of a distortion factor of the picked-up images taking place at the image pickup by the camera and performs a projective transformation of the picked-up image from which distortion is removed through the use of the distortion parameter to associate the transformed image with the base image.
Still moreover, it is also appropriate that the transformation parameter calculating means or step corrects the luminance value of the base image in accordance with the luminance values of the picked-up images. In this case, a preferred correction is possible by extracting, from the picked-up images, an area in which the luminance values are approximately equal to each other and obtains the average value of the luminance values in the extracted area to replace the luminance value of the corresponding pixel in the base image with the average value.
Still moreover, it is also appropriate that the image inputting means or step inputs a plurality of picked-up images taken by a plurality of cameras standing in a predetermined positional relationship.
Furthermore, a third aspect of the invention provides a computer-readable program providing medium for providing, in a material computer-readable form, a computer program for implementing, on a computer system, a camera calibration to calculate a parameter representative of a characteristic of a camera, with the computer program comprising an image inputting step of inputting a picked-up image obtained by taking a photograph of a pattern with a predefined geometrical configuration through the use of a camera, an image holding step of holding a base image comprising the pattern with the predefined geometrical configuration, and a transformation parameter calculating step of calculating a transformation parameter on the basis of the association (correspondence) in pixel between the picked-up image and the base image.
Still furthermore, a fourth aspect of the invention provides a camera capable of conducting a camera calibration, comprising image inputting means for inputting a picked-up image, image holding means for holding a base image comprising a pattern with a predefined geometrical configuration, and transformation parameter calculating means for calculating a transformation parameter on the basis of the association (correspondence) in pixel between the picked-up image, which has a pattern with the predefined geometrical configuration, inputted through the image inputting means and the base image.
According to the invention, the camera parameter calculation, i.e., the calibration, is made by the image registration between a picked-up image taken actually through a camera and a base image composed (or combined) in a computer.
The picked-up image for use in the calibration is inputted by taking a photograph of a calibration pattern whose geometrical configuration is known in advance, with the inputted picked-up image being stored temporarily, for example, in a frame buffer existing in the interior of a camera calibration apparatus.
Meanwhile, the base image having a pattern univocally corresponding in the definition of the geometrical configuration to the calibration pattern is retained in another frame buffer of the camera calibration apparatus. The camera calibration apparatus also can accept an arrangement in which a base image including a pattern created in advance is put in an external a storage device such as a hard disk and, when needed, is fetched from this disk to be written in a frame buffer.
Alternatively, it is also appropriate that, in the interior of the camera calibration apparatus, a base image is generated on the basis of the definition of a geometrical configuration according to a computer graphic technology, and is stored temporarily in a frame buffer. In brief, the base image written in the frame buffer has a complete pattern conform to the theoretical geometrical configuration.
The picked-up image, i.e., the pattern to be taken actually through a camera, is not required to be a pattern formed permanently on a plane by printing, but it is also acceptable if that pattern is developed by projecting a base image, generated with the computer graphics, on a plane by means of a light projecting means (for example, a slide). In this case, it is relatively easy to maintain the univocal or unique property in geometrical configuration of a calibration pattern between the base image and the picked-up image.
The parameter calculation can be made by performing the image registration between the picked-up image and the base image thus obtained and further by reducing the luminance error to a minimum.
Thus, the camera calibration apparatus and method according to the present invention enable stable high-accuracy parameter estimation using one picked-up image.
Accordingly, although the pattern to be taken by the camera is required to be univocal with respect to the geometrical configuration definition to be used for pattern composition for the base image in the computer, no limitation is imposed on the distance to the subject or the pattern size. This is because the projective transformation can take care of the scale of the photographed pattern involved in the variation in distance and the difference in photographing direction. The projective transformation can be made relatively easy through arithmetic processing in a computer resource.
Because of conducting the processing without using local characteristic points, the invention can not only eliminate the influence on the parameter estimation from an error generated at the extraction of the characteristic points, or the like, but also restrain the influence of a noise of an image taken through a camera. Additionally, the invention can accomplish stable parameter calculation through the use of one picked-up image.
In addition, a combination of simple figures (for example, checkers with two colors of black and white or a combination of binary triangles) functions satisfactorily as a pattern to be used for the calibration,. For a photograph of such a pattern, there is a need to prepare a plane having a pattern identical to the composed pattern, but there is no need to equalize the distance to the camera or the size of the pattern; therefore, the condition on the calibration is reducible. Additionally, for the calibration, it is also possible to pick up an image obtained by projecting a pattern, projected using a slide or the like, on a plain plane.
Still additionally, since a pattern already known, that is, a pattern with a predefined geometrical configuration, is used for the calibration, it becomes easy to construct a preprocessing algorithm for luminance correction or the like. The calibration can also be designed using a pattern which enables easy construction of a preprocessing algorithm.
The camera calibration according to the invention can calculate a projective transformation matrix indicative of the correspondence between a composed image and a picked-up image simultaneously with calculating a distortion parameter, and is also applicable to a calibration for the stereo-method. That is, in a manner that a coordinate transformation is made through a projective transformation for associating one of two picked-up images by a camera(s) with a base image and an inverse transformation of a projective transformation for associating the other picked-up image with the reference image, the association between the two picked-up images is obtainable.
For example, Japanese Patent Application No. 9-207948 or 9-207951, already assigned to this applicant, discloses an image processing apparatus and method based on the stereo-method, and the invention is also applicable to these image processing apparatus and method.
A program providing medium according to the third aspect of the invention is a medium for offering, in a material computer-readable form, a computer program to, for example, general-purpose computer systems capable of implementing various program codes. This medium can be a detachable portable storage medium such as CD (Compact Disc), FD (Floppy Disc) or MO (Magneto-Optical disc), or a transmission medium such as a network, and limitation is not particularly imposed on form.
Such a program providing medium defines a cooperative relationship in structure or function between a predetermined computer program and the providing medium for realizing the function of the computer program on a computer system. In other words, a predetermined computer program is installed in a computer system through a program providing medium according to each of seventh to ninth aspects of the invention to exhibit a cooperative operation on the computer system, thereby offering the same effects as those of the first aspect of the invention.
Other object and features of the present invention will become more readily apparent from the following detailed description of the preferred embodiments taken in conjunction with the accompanying drawings.